perf: make Hacktoberfest prep tracker fast + verbose (async fan-out, progress) - #15226
Merged
cclauss merged 1 commit intoSep 8, 2026
Merged
Conversation
The daily prep job spent ~16 minutes because it looked up the changed files of every open 'awaiting reviews' PR one request at a time. Those lookups are independent, so fire them concurrently through a single httpx2.AsyncClient bounded by a small semaphore, and likewise resolve the tracked-row PR states and the issue/PR counts concurrently. Report progress to stderr (flushed) so a human watching the Actions log can see the job is alive. Runtime drops from ~16 min of serial round-trips to well under a minute.
Closing this pull request as invalid@priya-sundaram-dev, this pull request is being closed as none of the checkboxes have been marked. It is important that you go through the checklist and mark the ones relevant to this pull request. Please read the Contributing guidelines. If you're facing any problem on how to mark a checkbox, please read the following instructions:
NOTE: Only |
cclauss
enabled auto-merge (squash)
September 8, 2026 07:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #15225, addressing @cclauss's feedback on #15081: the daily
hacktoberfest_preprun took ~16 minutes with no output.What was slow
scripts/hacktoberfest_prep_update.pylooks up the changed files of every openawaiting reviewsPR to rank the busiest directories — up to a few hundredGET /pulls/{n}/filescalls, previously issued one at a time. That serial chain, not the total request count, is what ate the 16 minutes.Changes
httpx2.AsyncClient(per the async guide) bounded by a smallasyncio.Semaphore(CONCURRENCY = 8) to stay well under GitHub's secondary rate limits. Tracked-row PR-state checks and the three issue/PR counts are gathered concurrently too. Runtime drops from ~16 min of serial round-trips to well under a minute....scanned N/totaltick every 25 PRs. The final line now reports elapsed seconds.main(); only the network work is async (keepsASYNC230happy).Rate-limit/5xx retry behaviour is unchanged, just moved into the async helper. Stdlib
asyncio+ the already-installedhttpx2, so no workflow change is needed.On the two links you shared
X-RateLimit-Reset/5xx backoff, which I kept.Testing
Ran the whole flow against a stubbed client (checkbox ticking for merged/closed vs still-open rows, stats block, top-3 directory ranking with correct singular/plural, progress lines, exit code).
ruff check/ruff formatclean.